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ABSTRACT 

By  perturbing  properly  a  linear  program  to  a  separable  quadratic  program 
it  is  possible  to  solve  the  latter  in  its  dual  variable  space  by  iterative 
techniques  such  as  sparsity-preserving  SOR  (successive  overrelaxations).  In 
this  way  large  sparse  linear  programs  can  be  handled. 

In  this  paper  we  give  a  new  conqautational  criterion  to  check  whether  the 
solution  of  the  perturbed  quadratic  programs  provide  the  least  2-norm  solution 
of  the  original  linear  program.  This  criterion  improves  on  the  criterion 
proposed  in  an  earlier  paper. 

We  also  describe  an  algorithm  for  solving  linear  programs  which  is  based 
on  the  SOR  methods.  The  main  property  of  this  algorithm  is  that,  under  mild 
assumptions,  it  finds  the  least  2-norm  solution  of  a  linear  program  in  a 
finite  number  of  iterations. 
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SIGNIFICANCE  AND  EXPLANATION 


This  paper  provides  the  theory  and  practical  algorithms  for  computing  the 
unique  smallest  solution  of  a  given  linear  program.  The  proposed  -ethod  is 
capable  of  handling  very  large  sparse  linear  programs  that  cannot  be  solved  by 
the  conventional  simplex  method. 


The  responsibility  for  the  wording  and  views  expressed  in  this  descriptive 
summary  lies  with  MRC,  and  not  with  the  author  of  this  report. 


A  FINITE  ALGORITHM  FOR  THE  LEAST  TWO-NORM  SOLUTION 
OF  A  LINEAR  PROGRAM 

Stefano  Lucidi* 

1 .  INTRODUCTION 

It  was  shown  in  [f]  and  [2j  that  the  least  2-norm  solution  of  a  linear 
program  can  be  computed  by  "properly"  perturbing  the  linear  program  to  a 
separable  quadratic  program  and  by  solving  the  latter  in  its  dual  variable  by 
iterative  techniques  such  as  SOR  methods  (see  [3]).  In  this  way  large  sparse 
linear  programs,  not  solvable  by  standard  pivotal  methods  can  be  handled  (see 
[41,  [1]).  The  main  difficulty  with  this  approach  is  that,  until  now,  there 
is  no  easy  way  to  know  "a  priori"  if  the  perturbed  quadratic  problems  is  a 
"proper"  perturbation  of  the  original  linear  program. 

In  order  to  overcome  this  difficulty  a  computational  criterion  to  check 
whether  the  solutions  of  the  perturbed  quadratic  program  was  given  in  [5]  and 
two  algorithms  where  the  perturbation  parameter  of  the  linear  program  is 
decreased  during  the  computational  procedure  were  proposed  in  [6]  and  [5] . 

We  give  now  an  outline  of  the  paper.  Section  2  contains  the  problem 
formulation.  In  Section  3  we  characterize  the  least  2-norm  solution  of  a 
linear  program  in  terms  of  the  constraints  which  are  active  at  that  point. 

This  characterization  will  be  utilized  in  subsequent  sections  of  the  paper. 

In  Section  4  we  turn  our  attention  to  the  problem  of  the  perturbation  of  a 
linear  program  and  we  propone  a  new  criterion  to  check  whether  the  solution  of 
the  perturbed  problem  is  in  fact  the  least  2-norm  solution  of  the  linear 
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program.  This  criterion  improves  on  that  given  in  [5].  In  the  last  section 
we  describe  an  algorithm  for  solving  linear  programs  which  is  based  on  the  SOR 
method.  The  main  property  of  this  algorithm  is  that,  under  mild  assumptions, 
it  finds  the  least  2-norm  solution  of  a  linear  program  in  a  finite  number  of 
iterations. 

We  briefly  describe  the  notation  used.  All  the  matrices  and  vectors  are 
real.  For  the  n  *  n  matrix  A  we  denote  row  i  by  A^ ,  column  j  by  A  ^ 
and  the  element  in  row  i  and  column  j  by  A^.  For  x  in  the  real  n- 
dimensional  Euclidean  space  r”,  x^  denotes  element  i  for  i  =  1,...,n 
and  x+  denotes  the  vector  with  components  (x+)i  =  maxCx^.O}  i  *  1,...,n. 

All  vectors  are  column  vectors  unless  transposed  by  T.  We  will  denote  the  2- 

T  V  n  1/~ 
norm,  (x  *)  ^  *  (  J  x 

j=*1 

orthant  {x  s  xeR0,  x  >  0} .  For  a  point  c  in  R°  and  a  closed  set  X  in 

Rn  the  2-norm  projection  p2(c,X)  of  the  point  c  on  X  is  defined 

lc  -  p_(c,X)l  =  mini  c-xl 
2  xe  X 

For  given  index  sets  I,  J  we  denote,  as  usual 

1 1 1  *  number  of  indices  in  I 
ru  «  fi  s  id,  ieJ> 

I\J  -  {i  s  id,  i/J] 

and  0 1  -j.  |  will  be  a  null  vector  of  dimension  |l|.  Finally  we  denote  the 

identity  matrix  by  E,  Aj  will  be  the  submatrix  of  A  with  rows  A^  i  c  I, 

Ajj  the  submatrix  of  A  with  elements  A^,  ie  I,  j  e  J  and  Xj  will 
denote  x^,  i  e  I. 


by  Ixl ,  while  R^  will  denote  the  nonnegative 


I 
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2.  PROBLEM  FORMULATION 


We  consider  the  linear  program 

T 

Max  c  x 


S.t.  Ax  <  b  (  1) 

x  >  0 

where  A  £  R®*n,  c  £  R°,  b  £  if  and  X  =  {x  :  Ax<h,  x>0}  +  ff.  Let  X  denote 
the  (possibly  empty)  optimal  solution  set  of  (1).  The  least  2-norm  solution 
of  (1),  p2(0fX),  can  be  characterized  as  the  unique  solution  of  the 

following  problem 

1  2 

Min 

S.t.  Ax  <  b 


cx  >  p 


where  p  is  the  maximum  value  of  ( 1 ) 


Given  any  x  e 

X,  u  c 

R®  and  v  c 

rP  we 

define 

the 

following  index 

sets : 

Ia(x) 

{i  : 

AiX  =  bi^  ' 

Ja(x) 

:=  {1 

:  Xi 

-  0} 

In(x) 

:*  {i  j 

Aix  <  hi^  ' 

dn(x) 

:=  {i 

:  xA 

>  0} 

Id(u) 

:=  { 1  : 

^  >  0}  , 

Jd(v> 

:=  U 

!  vi 

>  0} 

Let  Ib(x)  and  Jb(x)  denote  two  index  sets  such  that: 

i)  Ib(x)  C  Ia(x),  Jb(x)  C  Ja(x). 

T  T 

ii)  The  vectors  A^,  i  £  Ife(x)  and  E^ ,  i  £  Jb(x)  (where  E^  is  the 

i-th  row  of  the  identity  matrix)  are  linearly  independent. 

All  the  vectors  aJ,  i  c  Ia(x)\lb(x)  and  E^,  i  £  Ja(x)\jfe(x)  are 

T  T 

linear  combinations  of  vectors  A^f  i  e  Tb(x)  and  e£,  i  £  Jb(x). 


iii ) 


1 

i 


Furthermore  we  introduce  the  matrix  B(x)  c  R^^  and  the  vectors  q(x), 
s(x)  c  Rd,  where  d  =■  |l^(x)|  +  |jb(x)|,  as  follows: 

,  S(x)  :=  (B(x)B(x)T)  ^(x)  . 

(3) 

REMARK  1.  From  ill)  it  follows  that: 

Vi  €  Ia(x)\lb(x)  there  exists  a  z^  e  Rd  such  that 

Ai  =  B(x)  zi  ,  q(x)  zi  =  bt 

Vi  €  Ja(x)\J^(x)  there  exists  a  e  R^  such  that 

T  T —  T— 

E1  =  B(x)  zt  ,  q(x)  zt  =  0  . 


E  (x) 

0 !  j.  (x)  1 

b 

D 

B(x)  :» 

Aj  (x) 
h 

t  q(x)  :« 

bT  (x) 
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3.  PROPERTIES  OF  THE  LEAST  2-NORM  SOLUTION  OF  A  LINEAR  PROGRAM 

In  this  section  we  show  some  elementary  but  important  properties  for  the 
least  2-norm  solution  of  a  linear  program. 


PROPOSITION  1.  Let  x  be  a  solution  of  the  linear  program  (1)  and  (u,v)  be 
a  solution  of  its  dual.  Then  the  following  statements  hold: 

a)  If  x  «  P2(0,X)  then  x  is  also  the  least  2-norm  solution  of  the 
following  system  of  equations 


B(x)x  =  q(x) 


(4) 


that  is  x  *  B(x)T(B(x)B(x)T)  *q(x)  »  B(x)Ts(x) 

—  —  T  —  —  T  -1  — 

b)  If  x  «  B( x )  (B(x)B(x)  )  q(x)  +  Kc,  for  some  K  c  R,  and 


3(  X) 


Ib(x)\ld(u) 


<  0,  s ( x) 


Jb(x)\Jd(v) 


>  0  if  follows  that  x  **  PjCOjX). 


c)  Moreover,  if  the  gradients  of  active  constraints  are  linearly  independent 


at  x  then 

/ 

x  -  BT(x) (B(x)B(x)T)  q(x) 


s(x) 


Ib(x)\ld(u) 


<  0  ,  s ( x) 


>  o, 


<=*“*“>  X  -  P2(0,X)) 


Jb(x)\Jd(u) 


PROOF.  First  of  all  we  set 

I  I  (x),  J  J  (x),  I  :»  I  (x),  J  :»  J  (x),  I.  I.(u),  J.  J.(v)  , 
aa  aa  nn  nn  a  a  a  a 

Xb  s”  Ib^x)'  Jb  !“  Jb(x>'  B  !*  B(x),  q  :«  q(x),  s  3(x) 

»v 

a)  We  prove  this  part  by  contradiction.  Suppose  that  axists  x,  with 
x  /  x,  which  is  the  least  2-norm  solution  of  (4).  Then  all  the  points 
x(0)  «  0x  +  (1-0)x,  V0  €  R,  solve  (4)  and  from  the  maximum  principle  it 
follows 

xT(x(0 )  -  x)  -  0  .  (5) 

By  taking  into  account  Remark  1  we  have 
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Vi  C  J  \J.  x.  (0)  =  zTbx ( 6 )  =»  z^q  -  0 

a  d  l  i 

Vi  £  I  \l.  A . x (0  )  =  ?T3x(0)  =  zTq  =  b.  . 

a  d  x  i 

Hence  xT  (0)  =  0,  A-  x{0)  =  bT  ,  V0  £  R.  Since  x,  >0  and 
a  4  a  _  n  _ 

A,  x  <  b  there  exists  a  value  0  e  (0,1)  such  that  x_  (0 )  >  0, 

XX  J 

n  H  n  —  n 

A  x(0)  <  bj  .  Therefore  x(0)  e  X  and  it  is,  also,  an  optimal  point  for 
n  n 

(1):  in  fact  from  the  KXT  conditions  for  (1)  it  follows: 

T  —  T  _  — fp  —  —if  T— 

c  x  ( 0 )  *  Cj  x  ( 0 )  j  =  Uj  Aj  J  x(0)j  ■  Uj  bj  »  C  X 

n  n  aan  n  aa 

Hence  we  can  conclude  that  x(0)  is  feasible  for  problem  (2).  Now,  by 
applying  the  maximum  principle  to  problem  (2)  we  obtain 

x1’ (x(0)  -  x)  >  0  . 

Furthermore  by  (5)  we  have 

0  <xT(x(0)  -  x)  «  x^ (0x  +  ( 1-0 )x  -  x) 

*  F(xx  -  lx«2)  *  F(lx»2  -  »xl2)  <  0  . 

Therefore  x  is  the  least  2-norm  solution  of  (3)  that  is  (see  [7]) 

—  T  T  -1  T 

x«B(BB)  q=Bs 

b)  By  assumption  we  have 


X  ,  BTs  +  KC  -  A^ni^  Sj^  +  Aj  Sj  Xl 


T  T 

+  F‘ J  OJ  SJ  Oj 

b  d  b  d  b  d  b  d 


(6) 


and  s. 


\1A  <  °'  3JW\J 


b  d 


>  0. 


b  d 


From  the  KKT  condition  for  (1)  it  follows 

-T  *  -  et  * 

‘d  Ld  "d  ~d 


y(-c  +  Aj  v  )  =*  0 


VY  €  R 


(7) 
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Now,  by  adding  (6  and  (7),  we  obtain 


-  <Y«>=  *  Va(,Va  '  V^’  *  *  V'a'^V'a’ 

•  'WTx% *  v*.’ "  EW  V„  ■  Bv%  \«a  ‘  °  • 


If  we  choose  a  y  >  0  such  that 

Y+K  >  °'  Y\rvr  +  3 j  oj  >  YVoi  -  8i  ni  >  0 

bd  bd  bd  bd 

and  we  set 


u*  :=  Yu 


*bnid  '  ’  V*d  V^d  ' 


,  U*  .  :=  iu.  ,  T  ,  U*  .  :=  -S 


Vlb'  V1* 


Vrd  ' 


V'd  :=  YVJb™d +  Sjb™d '  Vvjb :=  YVJa'Jb '  Vvjd :=  svjd ' 


Y*  =  Y+K, 


ui  s-  °' 


v*  :=  0 
J 

n 


we  have  that  (x,y*,u*,v*)  satisfy  the  KKT  conditions  for  (2). 
c)  (==>)  It  follows  from  part  b) 

(<==)  By  the  assumption  that  the  gradients  of  the  active  constraints  are 
linearly  independent  at  x  we  have 


Ta  _  Tb  '  Ja  “  Jb  * 


From  part  a)  it  follows 


*  -  bT”  -  V  V  +  »i  u  -i  \i  *  >j  V  *  'j  \j  *3  ■  <8> 

dd  adad  dd  adad 
Now  x  solve  also  problem  (2)  and,  hence,  there  exists  u*  >  0,  v*  >  0,  y*  >  0 
such  that  (x,y*,u*,v*)  satisfy  the  KKT  conditions  for  (2),  namely: 


“  «T 
x  =  A_ 


Ai  (~ui  )  +  Ej  vj  +  Y*c 

a  a  a  a 


but  by  using  the  KKT  condition  for  (1),  that  is 


T  —  T  — 

c  -  Aj  uT  -  Ej  Vj 

d  d  d  d 
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we  obtain 


x  -T'' 


v ) +  ai  M  t-ui  m  ) +  bj  -  >vj  ) 

dd  d  adad  dd  d 


+  EJ  \J/J  \J.  ' 

adad 


VZ) 


—  T 


Then,  by  assumption,  we  have  that  the  columns  of  B(x)  are  linearly 


independent  and,  hence,  there  must  exist  only  one  vector  y  such  that: 
_  —  t 

x  =  B(x)  y.  Therefore,  from  (8)  and  (9)  we  obtain 


'l  *  -  ui 

d  d  d 


3J  “  VJ  "  ^VJ  ' 
d  d  d 


!i  \t  "  si  \t  -  \t  <  0 
b  d  ad  ad 


SJ  \J,  =  V7  \J_ 

b  d  ad  ad 


>  0 


and  these  last  two  inequalities  complete  the  proof  of  part  c). 


We  are  now  ready  to  establish  a  criterion  for  determining  when  the 
solution  of  the  perturbed  linear  program  is  indeed  the  least  2-norm  solution 
of  the  unperturbed  linear  program. 


4.  PERTURBATION  OF  LINEAR  PROGRAMS 

In  [1],  [2]  it  was  shown  that  the  least  2-norm  solution  of  the  linear 

program  (1)  can  be  obtained  by  solving  a  "proper"  quadratic  perturbation  of 
the  linear  progr.-u-..  In  particular  we  have  the  following  important  result. 


THEOREM  1.  Let  the  linear  program  (1)  feasible.  Then 


l)  i)  <x  /  H>  ==>  <3 


e«  >  °  :  P2*e---'  -r2 


(T'x>  “  P„<°'x>  for  all  e  e  (0,e  *]> 


ii) 


X  =  0 


vx*  *  p2(  0,X)y 


<==<3  e*  >  0,  X*  :  p  (|,x)  -  X*  for  all  e  £  (0,e*]> 


b) 


T  C  + 

sup  c  x  =  •  <==>  B  p 2 (— , X ) H  +  •  as  e  +  0 

X£X 


PROOF.  See  [2] 


We  can  note  that  p2<— ,X)  a^so  a  solution  of  the  problem 


£  T  T 
Min  —  x  x  -  c  x 

s .t .  Ax  <  b 
x  >  0 


(11) 


and  that  the  quadratic  programming  dual  (see  [8])  to  (11)  is 

IT  2  T 

Min  —  Ii  u  ■  v  ■  cl  +ebu 
2 

8.t.  (u,v)  >  0 

where  the  primal  and  dual  variable  x  and  (u,v)  are  related  by 

1  T 

x»— (-Au+v+c) 


(12) 


(13) 


Until  now  there  was  no  simple  way  of  determining  "a  priori"  the  parameter  e* 
of  Theorem  1.  The  next  result  which  gives  on  "a  posteriori"  computational 
scheme  for  e*  shows  why  it  is  difficult  to  compute  e*  "a  priori".  In  fact 
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we  can  note  that  e*  depends  strictly  on  the  least  2-norm  solution  of  (1)  (by 
means  of  the  vector  s(x),  defined  by  (3))  and  on  the  optimal  solution  of  the 
dual  of  the  linear  program  ( 1 ) . 


PROPOSITION  2.  Assume  X  ?  0.  Let  the  gradients  of  the  active  constraint^!  be 
linearly  independent  at  x  =  p2(0,X)  and  let  (u*,v*)  be  the  solution  point 
of  the  dual  of  the  linear  program  ( 1 ) .  Then  it  follows  that 

P2(f'X)  “  *  *  P2(0,X)  V  e  c  (0,E*] 

where  e*  is  given  by 


min  { [  min 

icticl  (u*) ,s(x' 
a 


u*J/s(x)i],  [  _ 


min 


ie{i£Jd(v*),s(x)1<0} 


-  vj/sfx)^} 
(14) 


where  s(x)  is  defined  by  (3)  and  Id(u#)  =  {i  :  uj  >  0}  and  Jd(v*)  » 

{i  :  vj  >  0}. 

PROOF.  We  recall  the  definitions  given  in  Section  2  and  we  set  Ift  :=*  Ia(x), 
Ja  :»  Ja(x),  In  Xn(x),  Jn  J/x),  Id  :»  Id(u*),  Jd  :=  Jd(v*),  s  :=  s(x). 
Now,  from  the  KKT  conditions  for  (1)  we  have 


-c 


A^"  U* 

Xd  Td 


v 

d 


* 

J 


d 


0 


(15) 


By  using  part  c)  of  Proposition  1  we  obtain 

t—  .T  T  _T  _T 

e(x  -  Aj  S  -  Aj  \J  Sj  w  -  EJ  3J  "  Ej  \j  SJ  \jJ 

dd  adad  dd  adad 


(16) 


where  sj  \i  *  0  and  sj  \j  *  0* 


a  'M 


a '  d 


Hence,  by  adding  (15)  and  (16),  we  obtain 


•* - ' * («!. - "i .)  * * c5j>  • ("-m )  ■  ° 


a  d 


d  d 


J  \  J  v  J  \  J. 
ad  ad 
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If  we  choose  e*  as  the  largest  value  of  e  such  that 


u5  -  es,  >0  ,  vt  +  es,  >  0 

Id  *d  Jd  Jd 


(that  is  if  £*  is  given  by  (14))  we  can  define  V  e  e  (0,e*] 

u(e)  =u*  -es  u(e)i  \t  :=  ~cai  \i  '  u(e)i  -  0  '  <17> 

add  ad  ad  n 

v(e)j  :=  v*  +  e8j  ,  v(e)jXj  eSj  Xj  ,  vU)j  s-  0  ,  (18) 

ddd  ad  ad  n 

Then  it  turns  out  that  (x,u(e ) ,v(e ) )  is  the  unique  solution  of  the  KKT 

conditions  for  problem  (11),  Ve  e  (0,e*]. 

Therefore  we  have  that 

x  =  P2(0,X)  =  p2(pX)  V  e  €  ( 0,e  *] 

and  that  (u(e),v(e))  is  the  unique  optimal  solution  of  the  dual  problem 

(12). 


In  order  to  overcome  the  inherent  difficulty  of  determining  e*, 

"a  priori"  a  practical  computational  criterion  was  proposed  in  [5]  to  check 
whether  the  solutions  of  the  perturbed  quadratic  programs  provide  the  least  2- 
norm  solution  of  the  original  linear  program.  This  criterion  required  two 
points  (Xi,u1,v1),  i  *  1,2,  that  satisfy  the  KKT  conditions  for  problem  (11) 
with  two  different  values  for  e. 

In  the  sequel  we  give  an  improved  new  criterion  that  requires  only  one 
point  (x,u,v)  which  satisfies  the  KKT  conditions  for  problem  (11).  However 
this  new  criterion  requires  a  matrix  inversion.  By  contrast,  no  matrix 
inversion  was  required  in  [5] . 

In  the  next  theorem  we  make  use  of  the  index  sets  I„(x),  J.(x),  i_(x), 
Jn(x),  Ib(x),  Jb(x),  Id(u),  Jd(v),  the  matrix  B(x),  and  the  vectors 
q(x)  and  s(x)  which  were  defined  in  section  2.  furthermore  we  assume  that 
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I  (x)  U  J  (x)  /  0  where  x  is  the  solution  of  (11),  because  otherwise  we 

A  A 

have  trivially  that  x  /  p2(0,X). 

THEOREM  2.  Let  x  be  a  solution  of  problem  (11)*  that  is  x  =  Pj^/X), 

(u*v)  be  a  solution  of  the  dual  problem  (12)  and  let  e*  be  the  optimal 

value  for  e  given  by  (14).  Then 

a)  /  _  __ 

/u  _  >  -es(x)  _  ,  v  _  >  e; 

/  Ib(x)  Vx)  Jb(x) 

\  [E-BT(x)(B(x)B(x)T)-1B(x)]c  -  YC 

for  some  y  <  1 


and  (u*,v*)  are  optimal  for  the  dual  of  the  linear  program  (1)  where 


(u  _  +  es(x)  _  ) 

Ib(x) 


Ib(x) 


Ib(x, 


1  -  Y 


Vx)X  Vx) 


Vx)Nlb(x) 

____ 


Zn<*> 


(v  _  -  es(x)  _  ) 


Vx) 


Vx) 


J  (x)\J.  (x) 
a  d 


Jb(x) 


1  -  Y 


Ja(x)\Jb(x) 


1  -  Y 


v« 


Vx) 


(19) 


0  . 


(20) 


b)  If  the  gradients  of  the  active  constraints  are  linearly  independent  at  x, 
then  it  follows  that: 


>  -es(x) 


>  es(x) 


Ia(x) 


I  (x)  J  (x) 

A  A 


Ja(x) 


[E-BT(X) (B(x)B(x)T)-1B(x)]c  -  0 


/  X  *  0 

/ 

0  <  e  <  e* 


N^X  »  P2(7'x)  ”  p2(0,xy 


> 


<**> 


and,  if  x  *  p2(0,X),  the  optimal  solution  (u*,v*)  of  the  dual  of  the 
linear  program  (1)  is  given  by  (19)  and  (20)  with  y  *  0,  Ia(x)  *  Ib(x)  an<J 
Ja(*)  =  Jb(x). 

c)  If  the  gradients  of  the  active  constraints  are  linearly  independent  at  x 
and  the  strict  complementarity  holds  for  the  linear  program  ( 1 )  at  the  least 
2-norm  solution  p2(0,X),  then  it  follows  that: 


PROOF •  Again  we  set  I,  :*  I  (x),  J  :=  J  (x),  I  :=  I  (x),  J  :«  J  (x), 

a  a  a  a  nn  nn 

Ih  :=  I^x),  Jb  :»  Jb(x),  1^  :«  Id(u),  Jj  :»  Jj(v),  B  :=>  B(x),  q  :»  q(x), 
s  :=  s(x). 
a)  Define 

y  :=  ex-c  . 

Since  the  point  (x,u,v)  satisfies  the  KKT  condition  for  problem  (11),  the 
point  (y,u,v)  satisfies  the  KKT  conditions  of  the  following  problem 


1  2 
Min  —  lyl 

Ay  <  eb  -  Ac 
y  >  -c  . 


(21) 


Therefore  y  is  the  optimal  solution  of  (21). 

By  repeating  the  same  steps  of  the  proof  of  part  e.)  of  Proposition  1  we 
have  that  y  is  the  least  2-norm  solution  of  the  following  system  of 


equations 


By  “  Eq  -  Be 


(22) 


In  fact,  suppost  not.  Then  there  exists  a  y,  y  +  y,  which  is  the  least  norm 
solution  of  (22).  Then  V0  the  points  y(0)  »  0y  +  (1-0)y  solve  (22)  and 
( recalling  Remark  1 ) 

Vi  €  Jg\^b  y^Q)  “  *^By(0)  -  z^(eq-Bc)  -  -c^ 

Vi  £  I  \t  A,  y(9)  -  z^By(0)  «  z^(eq-Bc)  -  eb  -A.  c  . 

a  d  i  1  1  11 


Moreover,  since  y_  >  -c,  and  A_  y  <  eb_  -  A_  c,  there  exists  a  value 

<J  «J  111 

n  __  n  n  n  _  n 

0  c  (0,1)  such  that  y(0)j  >  “cj  and  Aj  y(0)  <  ebj  -  Aj  c.  Therefore 

n  n  n  n  n 

y(0)  “  {y‘  :  Ay*  <  eb-Ac,  y'  >  -c} . 

Now,  by  applying  again  the  maximum  principle,  we  obtain 

o  <  y^yf^-y)  *  ^(y1  y-lyi2)  =  0(1  yl 2  -  lyi2)  <  o 

Hence  we  can  conclude  that  y  is  the  least  norm  solution  of  (22),  namely 

—  T  t  -1 

y  -  B  (BB  )  (eq-Bc) 

from  which 

—  T  T  -1  T  T  -1 

ex  -  [E  -  B  (BB  )  B]c  +  EB  (BB  )  q 

T  T  -1  T 

-  [E  -  B  (BB  )  B]c  +  £B  s  . 

Now  by  recalling  the  EXT  conditions  for  (11)  we  have 

—  T  —  T  — 

ex  -  c  +  Aj  Uj  -  Bj  Vj  -  0 

a  a  a  a 


from  which  we  obtain 


TT-1  T  T—  T  — 

[E  -  B  (BB  )  B]c  +  €B  S  -  C  +  Aj  Uj  -  Ej  Vj  “0 

a  a  a  a 

(E  -  BT(BBT)"1B]C  -  C  +  A^  <ux  +€SI  >  +  Ai  Uj 

d  b  b  a  b  a  b 
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By  assumptions  we  have  [E  -  BT(BBT)_1B]  ■  yc  with  some  Y  <  1 

uT  +  esT  >  0,  v_  -  esT  >  0 
*b  *b  Jb  Jb 

Therefore  if  we  set 


%  *  e\ 
1  -  Y 


U!  \!b 

UI  \I  -  TTY  '  u?  s=  0  ' 

a  b  n 


v* 

J*. 


1  -  Y 


V'7a 

VJb !“ 


,  v*  :=  0  , 

n 


we  have  that  (x,u*,v*)  satisfy  the  KKT  conditions  for  the  linear  program 


(1). 

b)  («»>)  The  proof  of  this  implication  follows  directly  from  part  a). 

(<»»)  By  using  the  KKT  conditions  for  linear  program  (1)  we  know  that 
there  exist  (u,v)  such  that 


'J*  ~  'f  /w 

-c  +  A  u  -  E  v  »  0 
II  J  J 

a  a  a  a 


from  which 


c  -  B 


\ 


Therefore  we  have 


T  T  -1  T  T  -1  T 

[E-B  (BB  )  B] c  *  [E-B  (BB  )  B] B 


M 

a 

\\) 


Now,  by  repeating  the  same  steps  of  proof  of  Proposition  2  and  by  using  (17) 
and  (18)  we  obtain  that 

uT  >  -es_  ,  v,  >  es_ 

I  I  J  J 
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c)  The  proof  of  part  c)  follows  from  the  proof  of  part  b)  and  the  proof  of 
Proposition  2  (see  (17)  and  (18)). 

■ 

REMARK  2.  Part  b)  of  Theorem  2  characterizes  the  solvability  of  a  linear 
program  in  terms  of  the  solvability  of  a  convex  quadratic  function  minimiza¬ 
tion  on  the  non  negative  orthant  (problem  (12))  without  any  "a  priori" 
assumptions  regarding  the  solvability  of  the  linear  program  ( 1 ) . 

We  now  turn  our  attention  to  computational  procedures  for  determining  the 
least  2-norm  solution  of  a  linear  program. 


5.  AN  ALGORITHM  WITH  FINITE  TERMINATION  FOR  LINEAR  PROGRAMMING 


The  quadratic  programming  problem  (12)  can  be  solved  by  using  a  sparsity¬ 
preserving  SOR  algorithm  introduced  in  {4] .  More  specifically  we  have  the 
following  algorithm  where  we  have  assumed  that  Aj  ^  0,  Vj  »  1,...,ra. 

LPSOR  Algorithm 

Choose  <u°,v°)  e  Rj+n,  <u  c  (0,2)  and  e  >  0. 

Having  (uk,vk)  compute  (uk+1,vk+1)  as  follows: 

vk  -  c)  +  *bj))  + 

k  —  1 ,  •  •  • ,  m 


k+1 


(  11 

(u. 


(»  (  £  *  f 


IAI2  j  1-1  1  1  l-j  * 1  l 

3  j>1 


U.  - 


k+1 


(vk  -  u>(-ATuk+^  +  vk  +  c) ) 


The  principal  and  computationally-distinguishing  features  of  this  SOR 
algorithm  are  that  it  preserves  the  sparsity  structure  of  the  problem  and 
require  only  simple  operations,  and,  hence,  very  large  problems  can  be 
tackled. 

We  refer  to  [2]  and  [4]  for  a  more  complete  description  of  sparsity¬ 
preserving  SOR  algorithm,  here  we  only  recall  the  following  convergence  result 
for  the  LPSOR  Algorithm 

PROPOSITION  3.  Assume  that  X  j*  JU  and  that  the  gradients  of  the  active 
constraints  of  the  linear  program  (1)  at  the  optimal  point  x  -  p2(0,X)  are 
linearly  independent.  Let  {(uk,vk)}  be  the  sequence  generated  by  the  LPSOR 
Algorithm.  Then 

a)  There  exists  a  real  positive  number  e*  such  that  for  each  e  c  (0,e*), 

r  If  V 

the  sequence  {(u  ,v  )}  converges  to  a  point  (u(e),v(e))  which  solves 
problem  (12)  and  the  corresponding  x(e)  determined  by  (13)  is  independent  of 


£  and  x(£) 


»  x  ■  p2(0,X) . 

b)  Moreover,  If  strict  complementarity  holds  for  problem  (1)  at  the  least  2- 
norm  solution  x  •  PjtO#*),  then  there  exists  a  real  positive  number  e**, 

£**  <  £*,  such  that  for  each  e  e  (0,e**]  there  exists  a  k*  such  that  for 
all  k  >  k*  the  following  hold: 

{i  :  u*  >  0}  -  {i  :  uj  >  0}  **  Ia(x) 

U  :  u{  =  0}  -  U  :  u{  -  0}  -  ln(7) 

{i  *  vj  >  0}  -  {i  *  >  0}  -  Ja(x) 

{i  :  v£  *  0>  =  {i  :  vj  -  0}  -  Jn(x) 

where  (u*,v*)  is  the  optimal  solution  of  the  dual  of  the  linear  program  (1). 
(Ia(x),In(x) ,Ja(x) ,Jn(x)  were  defined  in  section  2.) 

PROOF.  See  [5], 

In  applying  part  i)  of  Proposition  2  we  must  be  able  to  select  a  value  of 
e  such  that  e  <  e*.  In  order  to  ensure  that  e  <  e*  we  may  have  to  choose 
very  small  values  for  e  yield  a  very  slow  convergence  of  the  LPSOR  Algorithm 
when  applied  to  the  problem  (12). 

In  order  to  overcome  both  the  lack  of  a  practical  "a  priori"  selection 
procedure  for  the  parameter  e*  and  the  need  to  solve  problem  (12)  exactly 
many  times  for  a  decreasing  sequence  of  e  values  (until  one  of  the  criteria 
of  Theorem  2,  Corollary  1  or  Theorem  2  of  [5]  is  satisfied)  we  propone  an 
algorithm  which  is  based  on  the  results  of  the  preceding  sections 

The  key  idea  of  this  algorithm  is  similar  to  that  of  the  algorithms 
proposed  in  [5],  [6],  where  problem  (12)  is  solved  only  approximately  after 
which  £  is  decreased  in  a  prescribed  manner. 
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The  main  distinguishing  property  of  the  present  algorithm  is  that,  under 

mild  assumptions,  it  finds  the  least  2-norm  solution  of  the  linear  program  (1) 

in  a  finite  number  of  iterations.  The  algorithm  described  below  makes  use  of 

two  preselected  sequences  {ek}  and  {Nk}  such  that 

lim  ek  *»  0  and  lim  »  +“  .  (23) 

k+«  k-*» 


ALGORITHM  I 

Choose  (u°,v°)  e  R?J+n.  Having  (uk“ 1  ,vk_1 ) ,  (  dSv*)  is  obtained  by 

implying  iterations  of  the  SOR  algorithm  to  the  problem  (12)  with 

k  k— 1  k— 1 

e  »  e  and  by  using  (u  ,v  )  as  a  starting  point. 


THEOREM  3.  Assume  that: 

a)  The  linear  program  (1)  is  solvable. 

b)  The  gradients  of  the  active  constraints  are  linearly  independent  at  the 
optimal  point  x  -  p2(0,X). 

c)  The  strict  complementarity  holds  for  the  linear  program  (1)  at  the  optimal 
point  x  «  p2(0,X). 

Let  {uk,vk}  be  the  sequence  produced  by  the  algorithm  and  let 


I*1  :■  {i  :  uk  >  0}  ,  :*  {i  svk  >0}  ,  Bk 


(24) 


Then  there  exist  a  k  such  that  for  all  k  >  k  it  follows  that  the  matrix 
B  has  full  rank  and 


Xk  (Bk)‘rsk  -  x  -  P2(0,X) 

where 
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3*  [ B  (B  )  I 

PROOF.  By  using  Proposition  3  we  have  that  there  exist  a  e**  and  a  k* 
such  that  V  e  c  (0,e**]  and  V  k  >  k*  we  have  (by  recalling  the  definitions 
given  in  section  2) 

I*  *  I  (x)  -  I. (x) ,  Jk  -  J  (x)  -  J.(x) 
an  an 

and,  hence 

Bk  =•  Bk(x),  sk  «=  s(x) 

Therefore,  there  exists  a  k  large  enough  such  that  £k  <  e**  and  N*  >  k* 
and,  hence,  the  proof  of  theorem  follows  from  part  a)  of  Proposition  1. 


(25) 


REMARK  3.  Practically  we  do  not  need  to  compute  the  matrix  ( bN Bk)T)-1,  we 
only  have  to  solve  the  system 


(B  (B  )  )s 


Therefore  any  efficient  sparsity  preserving  method  fo^  solving  linear  systems 
of  equations  can  be  used. 


THEOREM  4.  Assume  that: 

a)  The  linear  program  (1)  has  a  unique  solution  x. 

b)  The  gradients  of  the  active  constraints  are  linearly  independent  at  the 
optimal  point  x. 

Let  {u  ,v  }  be  the  sequence  produced  by  the  algorithm  and  let 
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(26) 


I*'  «  { i  :  uk  >  0}  , 

I15  *  {t  s  uk  -  0} , 

Then  there  exist  a  k  such  that  for 
square  non  singular  matrix  and 


J*  s-  {1  :  vK  >  0} 
x 


k  >  k  the  matrix  A  ,  .  Is  a 

T<~k 


k 

x 


( *  )e^k^ 
I*JK 


x  . 


PROOF.  By  the  assumptions  a)  and  b)  we  have  that  the  linear  program  (4)  and 
its  dual  have  unique  optimal  solutions  and  that  the  strict  complementarity 
holds  (see  page  26  of  [11]).  Now,  by  using  again  Proposition  3  we  have  that 


there  exist  a 

e**  and 

k*  such  that 

V  e  £  {  0  ,£ ** ] 

and  V  k  >  k*  we  have 

Ik 

-  I  (x)  - 

d 

-  V"5' 

Jk  -  Ja(x)  - 

Jb(x)  «  Jd(u) 

~v 

^lc  — 

I* 

=  In(x)  - 

{i  ;  u^  «  0} , 

J  -  Jn(x)  « 

(i  :  v^  “  0} 

where  (u,v)  is  the  optimal  solution  of  the  dual  of  the  linear  program  (1). 
Therefore  for  values  of  k  large  enough  (such  that  £  <  £**  and 

V 

N  >  k*)  the  proof  of  theorem  follows  by  using  usual  arguments  of  the  duality 
theory  for  linear  programs  (see  [11],  page  45). 


In  Algorithm  I  different  stopping  criterions  can  be  used.  As  examples  we 
propone  two  algorithms.  In  Algorithm  II  the  stopping  criterion  is  based  on 
Theorem  2  and  Theorem  3  and  it  uses  the  properties  of  the  multiplier  function 
X(x)  for  the  linear  program  (1)  (we  refer  to  [9]  and  [13]  for  the  definition 
and  the  properties  of  the  multiplier  function  X(x),  here  we  recall  only  that 
X(»)  is  a  function  from  R°  to  Fp*+n  such  that  if  x  is  optimal  for  the 
1  near  program  (1)  X(x)  gives  the  optimal  solution  of  the  dual  of  the  linear 
program) . 


-21- 


Algorithm  III  ia  based  on  Theorem  4  and  the  duality  theory  for  the  linear 
programs . 

Again  Algorithm  II  and  Algorithm  III  make  use  of  the  preselected 
sequences  {ek}  and  {Nk}  such  that  (23)  holds  and  the  sequence 
{i  },  {j*},  {lk},  {J*}/  (Bk>  and  {s*}  given  by  (24),  (25)  and  (26). 

ALGORITHM  II 

Data:  (u°,v°)  c  R^+n 

Step  0:  Set  k  =  1. 

k  V  k 

Step  1:  Compute  (u  ,v  )  by  applying  N  iterations  of  the  SOR  algorithm 
to  the  problem  (12)  with  e  *  ek  and  by  using  (uk-\vk_^)  as 
starting  point. 

,  ,  k  k  k  k  .  k  k  r_  _k  k  k. T  -1  k_ 

Step  2:  If  B  has  full  rank,  u  >-es  ,v  >es  ,  [E-B  (B  (B  )  )  B  ] c 

IK  ik  jk  jk 

go  to  step  3;  else  go  to  step  5. 

Step  3:  Set  xk  »  (Bk)Tsk  and  Ak  -  A(xk). 

Step  4:  If  (xk,Ak)  satisfy  the  KKT  conditions  for  the  linear  program  (1) 
stop;  else  go  to  step  5. 

Step  5:  Set  k  »  k+1,  go  to  step  1. 

REMARK  4.  Step  2  is  not  necessary,  it  serves  only  to  reduce  the  numbers  of 

computations  of  the  multiplier  function.  If  the  computation  of  the  matrix 
k  lr  T  «•  1 

(B  (B  )  )  is  too  expensive  we  can  replace  step  2  with 

Step  2':  If  K  *  iM,  with  i  *  1,2,...  ,  go  to  step  3;  else  go  to  step  5. 
where  M  is  a  fixed  positive  number. 

COROLLARY  1.  Under  the  assumption  a),  b)  and  c)  of  Theorem  3,  Algorithm  II 
terminates  in  a  finite  number  of  iterations  and  the  produced  point  xv  is  the 


least  2-norm  solution  of  the  linear  program  (1)  and  Xv  is  the  optimal 
solution  of  Its  dual. 

PROOF.  The  proof  follows  by  repeating  similar  arguments  of  proof  of  Theorem  3 
and  by  using  part  c)  of  Theorem  2  and  the  properties  of  the  multiplier 
function. 


ALGORITHM  III 

Data:  (u°,v°)  e  R^*". 

Step  0 :  Set  k  “  1 . 

Step  1:  Compute  (u^.v^)  by  applying  iteration  of  the  SOR  algorithm  to 

the  problem  (12)  with  e  =  e*  and  by  using  (uk-1,vk  as  starting 

point . 


Step  2:  If  A  is  a  square  nonsingular  matrix  go  to  step  3;  else  go  the 


step 

5? 

k  .  -h  k 

,  T  -1 

Step  3: 

Set 

z  “  (A  k~k>  b  k'  W 

=  (Vk>  v 

I  J  I 

I  J  J 

Step  4: 

If 

<A~v~vzk  "  b~J 
I  J*  I 

v  T  k 

I  <  0,  w*  >  0,  (A  w 

I  J 

c  )  >  0  stop? 
J 


else  go  to  step  5. 


Step  5 :  Set  k  =  k+ 1 ,  go  to  step  1 . 


COROLLARY  2.  Under  the  assumption  a)  and  b)  of  Theorem  4,  Algorithm  III 
terminates  in  a  finite  number  of  iterations  and  if  (z^w''1)  is  the  produced 
point  and  if  (xv,uv)  is  defined  as  follows 
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Then  xv  is  the  unique  optimal  solution  of  the  linear  program  ( 1 )  and  uv  is 
the  optimal  solution  of  its  dual . 

PROOF.  Hie  proof  follows  from  Theorem  4  and  from  the  duality  theory  for 
linear  programs  (see  [11],  pages  45-50). 

For  brevity's  sake,  here  we  do  not  discuss  an  interesting  algorithm  which 
is  the  conjunction  of  Algorithm  I  of  this  paper  with  the  algorithm  proposed  in 
[6].  In  fact  such  an  algorithm  would  be  linearly  or  superlinearly  convergent 
under  assumptions  weaker  then  those  of  Theorem  3  and  would  also  have  finite 
termination  under  the  assumption  a)  -  c)  of  Theorem  3. 
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